{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/workshops/Idiomatic%20Programmer%20-%20handbook%203%20-%20Codelab%202.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idiomatic Programmer Code Labs\n",
    "\n",
    "## Code Labs #2 - Get Familiar with Training\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Handbook 3/Part 11: Training & Deployment\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Pretraining for Weight Initialization\n",
    "    2. Early Stopping during Training\n",
    "    3. Model Saving and Restoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretraining\n",
    "\n",
    "We are going to do some pre-training runs to find a good initial weight initialization. Each time the weights are initialized, they are randomly choosen from the selected distribution (i.e., kernel_initializer).\n",
    "\n",
    "We will do the following:\n",
    "\n",
    "    1. Make three instances of the same model, each with their own weight initialization.\n",
    "    2. Take a subset of the training data (20%)\n",
    "    3. Train each model instance for a few epochs.\n",
    "    4. Pick the instance with the highest valuation accuracy.\n",
    "    5. Use this instance to train the model with the entire training data.\n",
    "\n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import Sequential, optimizers\n",
    "from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n",
    "from keras.utils import to_categorical\n",
    "from keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "# Let's use the CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# Normalize the pixel data\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "x_test  = (x_test  / 255.0).astype(np.float32)\n",
    "\n",
    "# One-hot encode the labels\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "# Let's take a fraction of the training data to test the weight initialization (20%)\n",
    "# Generally, we like to use all the training data for this purpose, but for brevity we will use 20%\n",
    "x_tmp = x_train[0:10000]\n",
    "y_tmp = y_train[0:10000]\n",
    "\n",
    "# We will use this function to build a simple CNN, using He-Normal initialization for the weights.\n",
    "def convNet(input_shape, nclasses):\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', kernel_initializer='he_normal',\n",
    "                     input_shape=input_shape))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal'))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu', kernel_initializer='he_normal'))\n",
    "    model.add(Dense(nclasses, activation='softmax'))\n",
    "    return model\n",
    "\n",
    "# Let's make 3 versions of the model, each with their own weight initialization.\n",
    "models = []\n",
    "for _ in range(3):\n",
    "    model = convNet((32, 32, 3), 10)\n",
    "    # We will use (assume best) learning rate of 0.001\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=optimizers.Adam(lr=0.001), metrics=['accuracy'])\n",
    "\n",
    "    # Let's do the short training of 20% of training data for 5 epochs.\n",
    "    model.fit(x_tmp, y_tmp, epochs=5, batch_size=32, validation_split=0.1, verbose=1)\n",
    "    \n",
    "    # Save a copy of the model\n",
    "    # HINT: We are saving the in-memory partially trained model\n",
    "    models.append(??)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's pick the model instance with the highest val_acc and train it with the full training data\n",
    "# HINT: Index will be 0, 1 or 2\n",
    "models[1].fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.1, verbose=1)\n",
    "\n",
    "score = model.evaluate(x_test, y_test)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EarlyStopping\n",
    "\n",
    "Note that the training accuracy in the above example keeps going up, but at some point the validation loss swings back up and validation goes down. That means you are overfitting -- even with the dropout.\n",
    "\n",
    "Let's now look on how to decide how many epochs we should run. We can use early stopping technique. In this case, we set the number of epochs larger than we anticipate, and then set an objective to reach. When the objective is reached, we stop training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# Let's try this with a fresh model, and not care about the weight initialization this time.\n",
    "model = convNet((32, 32, 3), 10)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizers.Adam(lr=0.001), metrics=['accuracy'])\n",
    "\n",
    "# Set an early stop (termination of training) when the valuation loss has stopped \n",
    "# reducing (default setting).\n",
    "earlystop = EarlyStopping(monitor='val_loss')\n",
    "\n",
    "# Train the model and use early stop to stop training early if the valuation loss \n",
    "# stops decreasing.\n",
    "# HINT: what goes in the callbacks list is the instance (variable) of the EarlyStopping object\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.1, verbose=1, callbacks=[??])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Saving and Restoring\n",
    "\n",
    "Let's do a basic store of the model and weights to disk, and then mimic restoring the model from disk to in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "# Save the model and trained weights and biases.\n",
    "model.save('mymodel.h5')\n",
    "\n",
    "# load a pre-trained model as a different model instance (mymodel instead of model)\n",
    "mymodel = load_model('mymodel.h5')\n",
    "\n",
    "# Let's verify we really do that.\n",
    "score = mymodel.evaluate(x_test, y_test)\n",
    "print(score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Code Lab"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
